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(U//FOUO) DEEPDIVE 
Configuration Read Me 



(U) Overview 

(C)The purpose of this document is to provide procedures to configure an XKEYSCORE server as a 
DEEPDIVE server. DEEPDIVE can be defined as featuring a filter in front of the traditional XKEYSCORE 
processor (back-end). It is a Federated Query system that has a rolling buffer of all unfiltered data 
processed by XKEYSCORE. One query scans all sites. 

(U//FOUO) DEEPDIVE has two distinct functions. The Front End ingests various input types (e.g., .pcap, 
.sff, Ethernet, sdh, and sotf packets), sessionizes the data and promotes the data to the Back End. The 
backend can also ingest different input types and uses tools such as packet_splatter, xks_xfip (a Fast IP 
session izer), METTLESOME, PROMOTER (optional), defrag, and sotf_output. 

(C) The DEEPDIVE Back End performs strong (e-mail) and soft (content) selection and provides real-time 
tipping. It uses GENESIS Appid/Fingerprints which are updated hourly to all accessible field sites. An 
appid identifies a specific protocol and details of a session. Fingerprints flag sessions that meet specific 
criteria. 



(U//FOUO) DEEPDIVE Dataflow 

(U//FOUO) Data packets "enter" DEEPDIVE' s front-end, are processed and are fully sessionized before 
being passed to the back-end. The data is then analyzed, processed and released or stored as the 
mission dictates. 
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(U//FOUO) XKS Deep Dive can be configured differently at each site depending on the priorities of your 
mission. 
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(U//FOUO) Configuring DEEPDIVE for M1NUTEMAN in xks.config 

(U//FOUO) Use these configurations if your front-end system is outputting SOTF packets only to an XKS 
DEEPDIVE. If this is the case, then follow these steps to configure DEEPDIVE for the MINUTEMAN 
program only after XKEYSCO RE software has been installed: 

1. (U) Logon as the user oper. 

2. (U//FOUO) At the command line from within any directory, type viconfig and then press 
Enter. The xks.config file will open. 

3 . (U//FOUO) In the Signal Acquisition configuration section of xks.config , confirm: 

a. signal_acquisition_enable = yes : By setting this option to yes, 
signal_acquisition processes and any associated configurations will be added to 
proc_resources. 

b. signal acquisition on master = yes : This creates a 
signal acquisition base on the Master. 

c. have promoter = false : This indicates no promoter is configured for the system. 

d. splatter_hosts = [masterjiostname] 

In this case, master_host name is the actual hostname of the Master server. Setting 
splatter_hosts equal to ma ster_hostname indicates that the master is the only 
back-end host to receive the SOTF file (e.g., xksOl, xks02, etc.). 

4. (C) In the [signal acquisition] section of xks.config, type: 

signal acquisition [% :base ] = sigad = US -XXX, config = 
signal acquisition . config, front end only = False 

(C) In this case, the commas separate three options: 

* [%:base] = sigad = US-XXX : This creates a signal acquisition base 
process on each host in the XKS cluster and configures each to the US SIGAD (XXX) that 
is carrying the data. 

Important: (U//FOUO) On each host, do not forget to change maste r host name to 
the appropriate Master server hostname. 

* Config = signal acquisition * config : Sets the configuration file to 
$XSCORE_DIR/config/signal_acquisition/signal_acquisition.config 

* f ront_end_only = False : Indicates the host will act as both a front-end and a 
back-end host. 

5. (U//FOUO) Type : wql and then press Enter to save and exit xks.config. You will now 
co nf 3g ure sign al_acquis iti on . config . 
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(U//FOUO) Completing the Configuration of DEEPDIVE for M1NUTEMAN 

(U//FOUO) To complete the configuration of DEEPDIVE for MINUTEMAN, be sure to configure 
s ign al_ acquis iti on . con fig : 

1. (U//FOUO) At the command line from within any directory, type sa-config and then press 
Enter. This will take you to $XSCORE/config/signal_acquisition. 

2. (U//FOUO) Open sign ai_ acquis iti on . con fig, or create a file by that name if it does not already 
exist. This file will be used to configure several front-end processes for ingesting, sessionizing, 
and reassembling data. Each process is described in the following table. 

(U//FOUO) 



Front-End Processes 


What It's Called 


What ft Does 


What It Means 


Packet Splatter 


Ingests packets (from files, from 
the network, from a capture 
card) in a variety of formats. 


If it's a packet stream, it can 
probably be fed into a 
DEEPDIVE. 


xFip 


Fast reassembly of TCP/I Pv4 and 
UDP/IPv4 streams*. 


DEEPDIVE sessionizes 
everything before making a 
keep/drop decision. 


METTLESOME 


Reassembly of streams from less 
common protocol stacks. 




Promoter 


Rule-based filtering of 
reassembled sessions, based on 
keyword, country code or 
appid/fingerprint. 


DEEPDIVE intelligently 
chooses the most useful 
traffic for retention . 


Defrag 


Fully rebuilds sessions** 


Enough content available to 
do full decoding/document 
descent at the Back End 


*up to a 2S6K limit 
**up to a 10MB limit 



(U//FOUO) 

Note : (U//FOUO) In this Read Me, we will not address the Promoter. 



3. (U//FOUO) In the signal_acquisition.config r type/edit the following configurations for the 
processes identified in step 2: 

a. packet, splatter, -p 23000 — casenotation source in channel sri 
-i 22000 -t sotf --stats topic ps stats -v -n 4 

, isCritical=True , asRoo t=True 

b. xks xfip, -f 

% { SEQ ( $XSCGRE DIR/conf ig/misc/xf ip auto incf . cf } } , count=4 
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c. Mettle tcmalloc, -f 

% { SEQ ($XSCORE_DIR/config/misc/mettle# ,cf ) } , count = 4 

d. xks_defrag,“i %{ FOR T_IN_INC ( 24000) } -o 5040,count=4 

4. (U//FOUO) Type : wq! and then press Enter to save and exit signaiacquisit ion. config. 

5. (U//FOUO) Perform the following commands only after making changes to both 
s ign oi_ acquis iti on . config and xks . config : 

6. (U//FOUO) At the command prompt, type xks setup processes and then press Enter. 
This will create signal_acquisition_base on each host in the cluster. 

7. (U//FOUO) At the command prompt, type xks proc start and then press Enter. This will 
start the newly created processes. 



(U//FOUO) Configuring DEEPDIVE for FORNSAT 

(U//FOUO) Use these configurations if your front-end system is a TURNWEALTHY and outputting 
packets, packet bundles, and sessions to an XKS DEEPDIVE. If this is the case, then follow these steps to 
configure DEEP DIVE for FORNSAT: 

1. (U) Logon as the user oper. 

2. (U//FOUO) At the command line from within any directory, type vi config and then press 
Enter. The xks. config file will open. 

3. (U//FOUO) In the #[signai acquisition] section of xks. config, set the following configurations: 

a. signal acquisition enable = yes : By setting this option to yes, 
signal acquisition processes and associated configurations will be added to 
proc resources* 

b. signal acquisition on master = no : This will not create a 
signal acquisition base onthe Master. 

c. have promoter = false : This indicates no promoter is configured for the system. 

d. signal acquisition [% :base ] = 

casenotat ion=dynamic, conf ig=generic packet to bundle * config 
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(U/FOUO) In the case, the comma separates two options: 

* [%:base] = casenotation=dynamic : This configures the multiple 
signal acquisition base process on all the hosts in the cluster (%). 

• Config = generic packet to bundle . con fig : This sets the 
configuration file to 

$X SCO RE_DIR/con fig/signal a cqui sit ion /gene ricpackettobundle.config 

Important: (U/FOUO) If it does not already exist, you must create and configure 
generic packet to bundle . conf ig, See below, Configuring 
generic_packet_bundfe. config, for configuration insructions. 

4. (U//FOUO) Type : wql and then press Enter to save and exit xks. config. 



Configurin g genericpocketbundie. config 

Configuring DEEPDIVE for FORNSAT also requires that you setup the generic_pocket_bundie. config file: 

1. (U//FOUO) At the command line from within any directory, type sa-config and then press 
Enter. This will take you to $XSCORE/config/signal_acquisition. 

2. (U//FOUO) Open generic_pocket_bundie.config, or create a file by that name if it does not 
already exist. 

3. (U//FOUO) In the generic_pocket_bundie.config f type/edit the following configurations for the 
processes identified in step 2: 

a. sotf mux 2 , -i 5038 -o 34000 -s 5010, isCritical=True 

b. xks_xfip, “f 

% { SEQ ( $XSCORE DIR/conf ig/misc/xf ip generic packet to bundle au 
to inc# . cf ) } , isCrit ica l=True 

c. Mettle tcmalloc, -f 

% { SEQ ( $XSCORE DIR/conf ig/mi sc/mettle generic packet to bundle 
auto inct.cf) } , isCriti cal-True 

d. xks defrag 2,-i 5039 -o 5040 , isCritical=True 

4. (U//FOUO) Type : wql and then press Enter to save and exit generic packet bundie.config. 
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(U) Additional Processes 



(U//FOUO) Run these additional processes only after making changes to the configurations in 
xks.config 



1. (U//FOUO) At the command prompt, type xks rsync push config and press Enter. 
This sets pushes configuration changes out to the slaves. 

2. (U//FOUO) At the command prompt type xks setup processes and press Enter. Th is 
creates the signal acquisition ha se process. 

3. (U//FOUO) At the command prompt type xks proc start and press Enter. This will 
ensure all of the running processes pick up any configuration changes. 
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